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Abstract 

The  North  American  Aerospace  Defense  Command  (NORAD)  and  United  States  Northern  Command 
(USNORTHCOM)  (N-NC)  crew  training  has  been  hindered  by  an  inability  to  conduct  dynamic  training  and 
exercises  on  multiple  Command  and  Control  (C2)  systems.  There  was  no  common  simulation  injector  because  of 
stove-piped  acquisitions  and  legacy  interfaces  that  were  incompatible  with  the  Live- Virtual-Constructive  Toolkits. 
The  N-NC  Joint  Training  and  Exercise  Directorate  could  not  afford  traditional  replication  or  emulation  of  all  C2 
systems  and  their  data  sources,  nor  the  inevitable  sustainment  costs.  This  paper  presents  a  cost-effective  solution  to 
provide  dynamic  scenario  injection  into  multiple  C2  systems:  leveraging  server  and  desktop  virtualization 
technology  described  in  previous  I/ITSEC  papers. 

The  virtualization  process  transforms  stand-alone  systems  into  functionally  equivalent  virtual  machines  (VMs). 
Server  virtualization  technology  lets  multiple  VMs  run  as  guests  on  a  single  host,  and  a  host  can  support  VMs 
running  different  operating  systems.  This  allows  entire  processing  strings,  distributed  throughout  North  America,  to 
be  converted  into  VMs  on  a  single  server.  Because  the  VMs  inherit  the  fidelity  of  the  actual  processors,  their  outputs 
are  as  authentic  as  the  operational  systems.  These  VMs  feed  processed  simulation  event  data  into  actual  C2  systems 
or  equivalent  VMs.  Future  operational  system  upgrades  can  be  virtualized  and  then  replace  existing  VMs  without 
changing  this  infrastructure. 

Desktop  virtualization  technology  allows  users  to  run  multiple  VMs  in  separate  windows  on  a  common  display.  N- 
NC  exploited  desktop  virtualization  to  simplify  the  trainee  and  model  operator’s  workspaces.  They  can  view  and 
manage  multiple  VMs  with  one  monitor,  keyboard  and  mouse  (controlling  simulations,  lower  echelon  processing 
and  operator  interaction,  and  viewing  C2  workstation  displays). 

N-NC  successfully  leveraged  server  and  desktop  virtualization  to  overcome  training  shortfalls  with  authentic 
processing  and  display  of  dynamic  simulation  data.  This  approach  can  be  used  as  an  archetype  for  a  variety  of 
testing,  training,  or  operational  uses. 
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Leveraging  Virtualization  Technology  for 
Command  and  Control  Systems  Training 


BACKGROUND 

The  North  American  Aerospace  Defense  Command 
(NORAD)  and  United  States  Northern  Command 
(USNORTHCOM)  (N-NC)  Joint  Training  and 
Exercise  Directorate’s  Modeling  and  Simulation 
Branch  has  been  updating  the  simulation  injection 
capabilities  into  legacy  and  emerging  Command  and 
Control  (C2)  systems.  These  enhanced  simulation 
capabilities  will  allow  the  warfighters  to  better  train 
for  the  NORAD  and  USNORTHCOM  missions.* 

The  NORAD  and  USNORTHCOM  Command 
Center  (N2C2)  Training  Program  consists  of  four 
phases:  1)  Initial  Qualification  Training  (QT),  2) 
Mission  Qualification  Training,  3)  Certification,  and 
4)  Continuation  Training  with  NORAD  Regions, 
USNORTHCOM  Subordinate,  Component,  and 
Supporting  Commands,  and  Interagency  partners. 

QT  for  all  NORAD  mission  crews  is  conducted  in  an 
isolated  training  facility,  and  includes  hands-on 
interaction  with  a  limited  set  of  the  Command’s 
legacy  systems.  Following  QT,  the  students  transition 
to  On-the-Job  Training  in  the  operational  N2C2  and 
learn  to  use  their  complete  set  of  C2  systems.  N2C2 
Operations  is  not  staffed  with  extra  crews,  which 
could  allow  the  operators  to  go  to  an  off- site  location 
for  crew  proficiency  training  and  exercises.  This 
forces  crews  to  train  at  their  operational  workstations 
during  lulls  in  real-world  activities. 

Common  technical  solutions  for  this  training  would 
either  be  a  separate,  but  co-located,  training  system 
or  a  training  capability  integrated  into  the  operational 
system  (Simulation-over-Live).  Either  approach  may 
not  interfere  unduly  with  ongoing  operations,  or 
require  a  bigger  footprint  of  displays,  keyboards  and 
mice.  Figure  1  shows  the  overall  goal  of  conducting 
local  training  and  exercises  by  capitalizing  on  the 
Department  of  Defense  Service  Modeling  and 


NORAD  mission:  to  conduct  aerospace  warning, 
aerospace  control,  and  maritime  warning  in  the  defense  of 
North  America.  USNORTHCOM  mission:  partner  to 
conduct  homeland  defense,  civil  support,  and  security 
cooperation  to  defend  and  secure  the  United  States  and  its 
interests.  (N-NC  2011) 


Simulation  Training  Toolkits  to  stimulate  all  of 
NORAD  and  USNORTHCOM  missions  C2  systems 
(NORAD  2011). 


Sim-to-C2  System  View 

One  virtual  environment  „  rjs 

stimulates  all  C2  systems  ; 


Figure  1.  Simulations  to  N2C2  Missions  C2 
Systems  (N-NC  2010) 

The  initial  focus  of  the  Modeling  and  Simulation 
Branch  was  to  upgrade  the  Air  Domain  training  and 
exercise  capabilities.  Their  Initial  Qualification, 
Mission  Qualification  and  Certification  training 
historically  relied  on  scenarios  for  the  Air  Mission 
Evolution  (AME)  C2  system.  These  scenarios 
contained  events  ranging  from  a  hijacked  airliner  to 
mass  bomber  raids.  Because  of  the  high  cost  and 
lengthy  development  time,  there  were  a  limited  set  of 
these  scenarios,  and  they  were  static  -  replaying  the 
same  events  each  time.  Further,  AME  was  not 
designed  to  interface  with  dynamic  injections  from 
any  of  the  Services’  Joint  Live,  Virtual,  and 
Constructive  (JLVC)  Toolkit  simulation  capabilities. 
N2C2  crews  regularly  documented  the  lack  of 
dynamic  scenarios  as  a  training  shortfall. 

To  address  shortfalls,  the  Modeling  and  Simulation 
Branch  contracted  for  changes  to  the  Air  Warfare 
Simulation  (AWSIM)  Toolkit’s  database  and  to 
output  unique  messages  used  by  AME.  Also,  AME’s 
stove-piped  communications  system  required  the 
development  of  a  specialized  interface  adapter  to 
inject  simulation-tagged  data.  These  changes 
provided  a  path  to  conduct  local  training  with 
dynamic  scenarios,  and  met  the  objective  of  “Train  as 
you  Fight”  by  training  on  the  real  AME  system  in  the 
N2C2. 
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CHANGING  C2  SYSTEMS  AND  NEW 
CHALLENGES 

The  initial  training  enhancement  efforts  had  to  be  re¬ 
evaluated  when  NORAD  changed  the  priority  of  Air 
Domain  C2  systems  in  2011.  AME  went  from 
primary,  and  almost  exclusive  use,  to  a  tertiary  role. 
The  other  two  higher  priority  C2  systems  are  the 
Remote  Tactical  Air  Picture  (RTAP)  and  Global 
Command  and  Control  System  (GCCS). 
Operationally,  the  source  of  data  to  all  three  of  these 
C2  systems  is  from  the  Battle  Control  Systems 
(BCS),  located  at  the  four  NORAD  Sectors  in 
Alaska,  Canada,  Western  US,  and  Eastern  US.  The 
basic  source  of  the  data  did  not  change,  but  the  crews 
were  directed  to  use  different  systems  and  displays  to 
conduct  their  missions.  The  inability  to  locally 
stimulate  these  systems  with  scenario  data  became  a 
significant  training  limitation  in  both  the  IQT  facility 
and  N2C2. 

One  of  the  new  systems  is  the  Remote  Tactical  Air 
Picture  (RTAP).  This  C2  system  provides  NORAD 
and  USNORTHCOM  Headquarters  with  a  more 
detailed  and  timely  view  of  tactical  air  events.  The 
need  for  a  tactical  view  is  because  of  the  post-9/11 
homeland  defense  missions.  Domestic  events  (i.e., 
hijacking  or  entering  restricted  airspace)  often  have 
shorter  timelines,  compared  to  strategic  bomber 
attack  by  long-range  aircraft.  However,  RTAP 
cannot  accept  local  injection  of  simulation-tagged 
scenario  data  and  is  too  complex  to  emulate.  The 
option  to  purchase  a  separate  training  suite  was  cost 
prohibitive,  and  simply  could  not  fit  into  the  existing 
server  room. 

The  second  priority  suite  of  C2  tools  includes  Global 
Command  and  Control  System  (GCCS),  its 
Integrated  C4I  System  Framework  (ICSF)  display 
client,  and  an  associated  Track  of  Interest  (TOI) 
Tracker.  GCCS  is  the  Common  Operational  Picture 
and  is  used  to  share  data  with  other  Combatant 
Commanders.  TOI  Tracker  extracts  some  information 
from  the  GCCS  server  database  to  populate  a  new 
“decision  support”  information  display. 

With  re-prioritization,  the  new  challenge  was  to 
provide  realistic,  dynamic  training  with  events 
synchronously  delivered  to  these  three  C2  systems  - 
while  constrained  to  the  same  operator  workstation 
footprint  in  the  command  center. 

The  Modeling  and  Simulation  Branch  met  this 
challenge  by  taking  advantage  of  the  common  data 
source,  BCS.  We  shifted  away  from  AWSIM 
emulating  the  inputs  into  the  AME  C2  systems. 


Instead,  we  focused  on  using  JLVC  Toolkit  to 
stimulate  a  BCS  with  simulation  data,  and  sending 
outputs  from  its  processing  into  our  systems.  With 
this  approach,  the  BCS  processing  and  outputs  would 
be  authentic,  and  BCS’s  simulation-tagged  outputs 
become  the  inputs  to  RTAP,  GCCS  and  AME. 

To  afford  the  proposed  authentic  processing,  we 
exploited  server  and  desktop  virtualization 
technologies.  Server  virtualization  provided  a  way  to 
basically  replicate  dozens  of  separate  systems  onto  a 
set  of  servers  that  fit  into  half  a  rack  of  equipment. 
Desktop  virtualization  enabled  us  to  display  the 
multiple  C2  training  system  views  on  a  single,  shared 
display  -  much  like  the  “windowed  views”  of  several 
applications  or  documents  on  a  personal  computer. 

VIRTUALIZATION  TECHNOLOGY 

There  are  numerous  white  papers,  case  studies,  and 
vendor  descriptions  of  server  and  desktop 
virtualization  technology.  Virtualization,  and  some  of 
its  benefits  and  limitations,  was  described  in  the 
I/ITSEC  2011  Software  for  the  Rest  of  Us  session’s 
“Next  Generation  of  Distributed  Training  utilizing 
SOA,  Cloud  Computing,  and  Virtualization” 
(Lanman,  J.T.,  Horvath,  S.D.,  Linos,  P.K.  2011). 

Server  virtualization  software  can  be  used  to 
transform  or  “virtualize”  the  hardware  resources  of  a 
computer  to  create  a  fully  functional  virtual  machine 
that  can  run  its  own  operating  system  and 
applications  just  like  a  “real”  computer. 
Virtualization  works  by  inserting  a  thin  layer  of 
software,  a  “hypervisor,”  directly  on  the 
virtualization  server  hardware  to  translate  the 
processing  needs  of  the  virtual  machine.  The 
leveraging  aspect  of  virtualization  is  that  the 
hypervisor  can  manage  multiple  VM  guests  running 
concurrently  on  a  single  physical  server.  It  allocates 
hardware  resources  dynamically  and  transparently  so 
each  virtual  machine  is  essentially  self-contained, 
eliminating  potential  conflicts.  Another  feature  of  this 
isolation  is  that  the  hypervisor  can  interface  with 
VMs  running  different  operating  systems,  such  as 
Windows  and  Linux.  With  server  virtualization, 
multiple  VMs  with  different  operating  systems  and 
applications  can  run  at  the  same  time  on  a  single 
server  or  spread  across  a  pool  of  servers,  with  each 
VM  having  access  to  the  resources  it  needs  when  it 
needs  them.  (VMWare  2012) 

It  is  important  to  note  that  server  virtualization  is 
about  consolidating  the  processing  of  multiple 
machines  onto  a  server  -  it  does  not  include  the 
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separate  monitors,  keyboards,  mice  and  other 
external  components.  An  exception  to  this  is  the 
virtualization  of  network  switches.  Virtualized 
switches  allow  VMs  that  would  normally  be 
interconnected  on  a  local  or  wide  area  network  to  be 
hooked  together  internally  on  the  same  physical 
server. 


inside  its  window;  just  as  internet  users  view  and 
interact  with  several  web  sites  in  different  tabs. 


TRAINING  AND  EXERCISE  ARCHITECTURE 
COMPONENTS 


Also,  because  the  hypervisor  isolates  the  virtual 
machines  from  the  physical  hardware,  we  are 
anticipating  an  easier  upgrade  to  the  next  major  BCS 
software  release  than  the  actual  operational  systems. 
The  next  BCS  version  requires  all  the  physical 
computers  to  be  upgraded  to  a  newer  generation  of 
servers,  whereas  our  architecture  should  not  require 
hardware  upgrades. 

The  two  hypervisor  licenses  we  used  were  from 
VMWare  and  Red  Hat.  VMWare  was  chosen  because 
of  its  industry-leading  role  and  product  maturity.  Red 
Hat  provided  cost-effective  licensing  for  the  majority 
of  systems  we  were  targeting  to  become  VMs.  (Red 
Hat  Enterprise  Linux  Advanced  Platform  license 
allows  an  unlimited  number  of  virtual  guests  per  host 
server  to  run  Red  Hat  as  their  operating  system.) 
(Red  Hat  2009) 

For  our  training  and  exercise  use,  we  needed  to  create 
an  end-to-end  string  of  virtual  machines  that 
represented  everything  from  the  stimulus,  through  the 
Sector  and  local  C2  mission  processing,  and  on  to  the 
crews’  client  or  workstations,  as  shown  in  Figure  2. 
The  final  stage  was  to  allow  crews  to  see  and  control 
the  client  VM,  since  unlike  a  physical  PC,  virtual 
machines  do  not  have  direct  connection  to  a 
keyboard,  display  video  or  mouse  (KVM). 


As  shown  in  Figure  2,  the  dynamic  stimulus  in  our 
architecture  is  a  virtualized  AWSIM  from  the  JLVC 
Toolkit.  This  gives  the  capability  to  inject  the  two 
necessary  types  of  scenario  data  into  a  virtual  BCS. 
One  of  the  data  feeds  from  AWSIM  is  synthetic  radar 
data  to  -  simulating  radar  data  from  the  Federal 
Aviation  Administration,  Transport  Canada,  or  the 
U.S.  or  Canadian  military.  The  other  type  is  Tactical 
Data  Link  (TADIL)  information,  representing  data 
from  military  airborne  warning  systems  or  interceptor 
aircraft. 

The  virtualized  BCS  is  made  up  of  components 
normally  deployed  on  several  single-purpose 
computers,  as  shown  in  Figure  3.  (Thales-Raytheon 
Systems  Company  LLC  2008)  At  the  NORAD 
Sectors,  the  main  processing  is  by  the  BCS  Sentry 
Real-Time  server.  Dozens  of  separate  operator 
workstations  are  used  to  perform  tasks  such  as 
surveillance  and  interceptor  control.  For  the  training 
system,  we  virtualized  the  Sentry  Real-Time  server 
and  two  of  the  operator  workstations,  and  connected 
them  on  their  own  internal  network.  A  virtualized 
RTAP  server  replicates  the  physical  version  of  a 
Sector  BCS’s  new  RTAP  server.  One  other  necessary 
BCS  component  is  a  protocol  translator  for  changing 
AWSIM  radar  data  from  a  Common  Digitizer-2  (CD- 
2)  format  into  BCS’s  internal  ASTERIX  format. 


Modelingand  Simulation  BCS  Operator  N2C2Crew 

OperatorjWhite  Cell)  (White  Cell)  (Trainee) 


Figure  2.  End-to-end  String  and  Participants 

Viewing  a  virtual  machine’s  display  presented  a 
different  set  of  challenges.  Desktop  virtualization 
provides  several  techniques  that  show  the  display,  or 
displays  of  multiple  VMs.  Thin  clients  act  as  “dumb 
terminals”  allowing  the  user  to  have  keyboard  and 
mouse  interaction  with  one  or  more  VMs  while 
showing  their  individual  displays  in  separate 
windows  on  a  common  monitor.  Alternatively,  the 
user  may  view  different  VMs  through  a  browser, 
such  as  Internet  Explorer.  Display  virtualization 
allows  the  user  to  run  several  VMs  in  parallel  - 
taking  control  of  any  individual  VM  by  clicking 


System  Block  Diagram 


Figure  3.  BCS  Block  Diagram  with  RTAP  Server 

VMs  are  basically  files.  Creating  a  duplicate  VM  is 
as  simple  as  duplicating  the  file,  and  giving  it  a 
unique  VM  name,  network  name  and  IP  address.  So, 
creating  the  second  operator  workstations  mentioned 
above  was  a  relatively  simple  duplication  task.  All  of 
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these  BCS  components  run  on  VMs  with  Red  Hat 
Linux  as  the  Operating  System. 

Because  the  virtual  AWSIM  and  virtual  BCS 
components  are  integrated  onto  a  single  virtualization 
server,  an  internal  network  switch  connection 
replaces  long-haul  communications.  This  direct 
connection  is  functionally  equivalent  to  the  real- 
world’ s  multiple  communications  interface  boxes, 
crypto  gear  and  circuits. 

The  output  from  the  virtual  BCS  is  also  locally 
available  on  internal  networks  on  the  virtualized 
platform.  This  eliminated  the  need  for  more  external 
communications  boxes  from  the  BCS  RTAP  directly 
to  the  virtualized  receiving  RTAP  Web  Server. 
However,  the  other  two  receiving  systems,  AME  and 
GCCS,  needed  minor  interface  adaptors  to  complete 
their  respective  connections. 

The  virtualized  RTAP,  representing  the  primary  C2 
system  used  at  the  Headquarters,  is  also  made  up  of 
multiple  components.  The  real-world  system  has  an 
RTAP  Web  Servers  that  normally  interface  long-haul 
to  the  RTAP  servers  at  the  NORAD  Sectors.  This 
Web  Server  pushes  data  to  the  Operators’  RTAP 
Workstations.  In  our  training  and  exercise 
environment,  the  Web  Server  and  all  the  RTAP 
Workstations  were  converted  into  a  set  of  virtual 
machines  on  their  own  private  network  switch.  As 
noted  above,  creating  eight  RTAP  Workstations  was 
relatively  easy;  duplicating  the  RWS1  Virtual 
machine  files,  renaming  them  RWS2  through  RWS8, 
and  assigning  each  a  unique  network  name  and  IP 
addresses. 

The  second  output  from  our  virtual  BCS  is  the 
TADIL  data  to  GCCS.  This  data  stream  is  handled 
somewhat  differently  from  the  others  because  we 
chose  to  re-use  a  separate  hardware  platform  for  the 
GCCS  server.  The  GCCS  server  was  not  virtualized 
because  the  commercial  virtualization  industry  did 
not  have  a  hypervisor  for  older  Sun  Microsystems 
hardware.  The  server  was  essentially  “no  cost”  - 
basically  pulled  out  of  the  recycling  bin  with  GCCS 
software  re-installed.  Also,  the  TADIL  interface  from 
BCS  to  GCCS  needed  a  minor  format  change.  The 
reformatting  is  done  using  a  Common  Boundary 
System  (CBS)  GOTS  tool  that  normally  runs  on  its 
own  stand-alone  computer.  The  CBS  application  is 
installed  on  a  virtualized  version  of  a  generic 
Windows  PC. 

Operationally,  the  ICSF  clients  that  display  GCCS 
TADIL  track  data  are  normally  individual  PCs.  In  our 


architecture  the  eight  clients  were  converted  to  VMs 
running  a  Windows  OS  with  the  ICSF  software 
application.  Because  the  ICSF  clients  are  on  the 
virtual  server,  we  have  another  external  network 
connection  from  the  separate,  physical  GCCS  server 
back  into  the  virtualization  server.  The  same  external 
connection  allows  TOI  Tracker  to  access  the  GCCS 
database. 

The  third  output  from  the  virtual  BCS  is  data  to 
AME.  Because  of  strict  limitations  on  injecting 
simulation  data  into  AME,  that  data  stream  is 
converted  to  a  special  “Service  Oriented  Scenario 
Injector”  format.  This  reformatting  is  also  done  by 
the  same  Common  Boundary  System  tool. 

For  a  more  robust  training  capability,  we  chose  to 
provide  the  capability  to  exercise  with  air  tracks 
flying  from  Sector  to  Sector.  This  allows  crews  to 
practice  their  “Cross-Border  Operations”  procedures. 
This  necessitated  creating  three  more  complete  sets 
of  virtual  BCS  systems,  so  we  could  model  all  four 
NORAD  Sectors.  The  replication  task  was  somewhat 
more  difficult  than  cloning  individual  VMs  because 
of  Sector-specific  naming  data  imbedded  inside  the 
BCS  subsystem  component  adaptation  files. 

The  expanded  architecture  (only  showing  two  of  the 
four  virtual  BCS  systems)  is  depicted  in  Figure  4.  A 
modification  to  AWSIM  was  needed  in  areas  of 
overlapping  radar  coverage  to  insure  that  synthetic 
radar  data  was  sent  to  both  affected  BCSs.  Note  that 
the  adapters  between  the  individual  virtual  BCS 
outputs  to  GCCS  and  AME  also  serve  as 
consolidation  nodes  for  combining  outputs  from  the 
separate  BCSs  into  merged  inputs  to  the  C2  systems. 
The  adapters  to  AME  and  GCCS  were  combined  into 
a  single  Windows-  based  virtual  machine  along  with 
the  TOI  Tracker  application,  and  labeled  “Common 
Boundary  System”  in  Figure  4. 

The  virtualized  systems  described  above  all  reside  in 
the  server  room.  But  these  virtual  machines  do  not 
have  direct  connections  to  displays.  Students  in  the 
Qualification  Trainer  and  crews  in  the  N2C2  need  to 
“see”  the  simulate  event  data.  Additionally,  the 
Modeling  and  Simulation  Operator  needs  to  “see”  the 
AWSIM  VM  in  order  to  control  the  dynamic  scenario 
inputs.  A  “White  Cell”  player  needs  to  “see”  BCS 
Workstations  to  act  as  the  BCS  Operator  and  execute 
essential  actions  in  reaction  to  an  event.  We  used  a 
combination  of  two  technical  solutions  for  viewing 
the  virtual  machines;  installing  new  thin  clients,  and 
re-using  Internet  Explorer  on  existing  workstations. 
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Figure  4.  Training  and  Exercise  Virtual  and  Physical  Architecture 


DEPLOYMENT  AND  USES 

There  are  three  virtualized  Air  Domain  training  and 
exercise  capabilities;  at  the  NORAD  and 
USNORTHCOM  Command  Center,  the  Qualification 
Trainer,  and  the  Canadian  Forces  Air  Warfare  Centre. 

The  N2C2  Air  Domain  training  capability  is 
integrated  into  the  operational  command  center.  This 
imposes  a  limitation  not  to  interfere  with  monitoring 
real-world  operations,  and  constrains  the  display  of 
the  training/exercise  information.  The  deployment 
reuses  the  existing  AME  exercise  capability  and  its 
display.  RTAP,  ICSF  and  TOI  Tracker  are 
“windows”  on  a  single  monitor  driven  by  a  new  thin 
client.  Figure  5  shows  the  AWSIM  Operator  driving 
scenario  data  into  the  virtual  BCS,  then  on  to  AME 
for  processing  and  display.  In  parallel  the  virtual  BCS 
sends  data  to  RTAP  and  GCCS  and  their  respective 
displays. 


Figure  5.  Scenario  Injection  through  to  N2C2 
Operator  Displays 

The  Qualification  Trainer  has  a  full  end-to-end 
capability  to  stimulate  the  RTAP,  GCCS  and  AME 
C2  systems  for  hands-on  training.  The  QT’s  systems 
were  intentionally  designed  to  be  isolated  and  have 
no  capability  to  connect  to  other  operational  systems 
or  networks.  There  is  no  competition  for  space  on 
their  display  because  there  is  no  requirement  for 
students  to  concurrently  monitor  real-world 
operations.  The  three  C2  system  views  can  be  spread 
across  several  monitors  through  a  browser  on  the 
existing  AME  workstation  and  a  new  thin  client. 

The  Canadian  Air  Warfare  Center  utilizes  a  scaled 
back  version  of  the  training/exercise  capability 
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because  they  are  only  required  to  support  the  single 
Canadian  Sector,  rather  than  the  four  Sectors  that 
supply  data  to  NORAD  Headquarters.  Their  system 
can  function  as  both  a  training/exercise  tool  on  the 
Canadian  Forces  Experimentation  Network,  or  as  a 
laboratory  and  development  suite. 


SAVINGS 

An  “apples-to-apples”  savings  comparison  to 
acquiring  a  set  of  equivalent  environments  on 
physical  systems  could  not  be  easily  estimated. 
Virtualization  gave  us  the  additional  flexibility  to 
easily  expand  to  a  full  breadth  and  depth  to  simulate 
four  NORAD  Sectors.  Our  training  and  exercise 
environment  can  dynamically  stimulate  the  three  C2 
systems.  At  the  low  end,  the  costs  in  Table  1  show  a 
baseline  configuration  with  eight  clients.  This  may 
not  be  representative  of  other  potential  virtualization 
activities  because  the  principle  software  components 
and  some  hardware  were  government  off-the-shelf 
(GOTS)  software  or  government-furnished 
equipment  (GFE).  The  costs  are  conservatively  a 
factor  of  ten  lower  than  estimates  for  a  comparable 
version  if  built  on  separate  physical  servers  and 
clients. 


Table  1.  QT  Facility  H/W  and  S/W  Costs 


Item 

Unit  Cost 

Number 

Total 

Dell  PowerEdge  R710 

$  8,387 

3 

$  25,161 

Red  Hat  Enterprise  Virtualization  for  Servers 

Red  Hat  Enterprise  Virtualization  for  Desktops 

$  499 

$  375 

4 

1 

$  1,996 
$  375 

Red  Hat  Enterprise  Linux  (RHEL)  Standard 

$  1,999 

2 

$  3,998 

VMware  vSphere  Essentials 

$  611 

1 

$  611 

Windows  2003  Server 

$  150 

1 

$  150 

Cisco  SF300-24  Switch 

$  215 

2 

$  430 

HP  5740  Thin  Client 

$  550 

8 

$  4,400 

Total 

$  32,782 

GCCS-J  ICSF  Client  S/W  (Win  OS) 

GOTS 

8 

Common  Boundary  System  S/W  (Win  OS) 

GOTS 

1 

BCS  S/W  components  -  NORAD  Sector  (RHEL  OS) 
RTAP  S/W  components  -  NORAD  HQ  (RHEL  OS) 

GOTS 

GOTS 

4 

1 

AWSIM  S/W  (RHEL  OS) 

GOTS 

1 

TOI  Tracker  (Win  2003  ServerOS) 

GOTS 

1 

Sun  V240  GCCS  Server  (Surplus) 

GFE 

1 

<  $1000 

Rack  (Surplus) 

GFE 

1 

<  $1500 

Misc  H/W  -  Cables,  tie  downs 

GFE 

<  $80 

An  often  asked  question  regarding  server 
virtualization  is  how  many  physical  systems  can  be 
run  as  virtual  machines  on  a  server.  There  are 
numerous  variables  that  have  significant  influence 
(processor  cores/socket  counts  and  speed,  RAM,  and 
disk  size/spindle  speed),  in  addition  to  the  operator’s 
actual  utilization  and  workload  demands. 

We  contracted  with  the  AME  developer  to  conduct  a 
virtualization  study  and  make  enhancements  to  the 
QT  facility.  As  part  of  the  study,  they  did  some 
quantitative  analyses  on  the  performance  of  physical 


systems  versus  virtualized  machines  running  the 
same  applications  and  under  the  same  loads  and 
operator  interactions.  Table  2  shows  the  density 
expectations  if  VMs  ran  on  platforms  approximately 
equivalent  to  those  that  hosted  the  stand-alone 
applications  and  workstation  functionalities.  The  data 
shows  an  ability  to  host  twenty  times  as  many  virtual 
application  servers  or  ten  times  the  number  of  virtual 
workstations  -  a  considerable  savings  of  equipment 
costs  and  associated  power,  HVAC,  and  rack  space. 

Table  2.  VM  Performance  and  Capacity  Analysis 

Application  Server  Thick  Client  Workstation 


Physical 

Virtual 

Peak  CPU 

1175.2  Mhz 
13% 

4374  Mhz 
41% 

Average 

723.2  Mhz 

2129.9  Mhz 

CPU 

8% 

20% 

Expected 

Density 

1 

-10  VMs 

Physical 

Virtual 

Peak  CPU 

3072  Mhz 

1836  Mhz 

24% 

17% 

Average 

1408  Mhz 

894.7  Mhz 

CPU 

11% 

8% 

Expected 

Density 

1 

-20  VMs 

Expected  Density  assumes  40%  CPU  reserved  for 
Expected  Density  assumes  40%  CPU  overhead  and  future  growth 

reserved  for  overhead  and  future  growth  •*  increase  in  Virtual  WS  CPU  is  due  to  overhead  of 

processing  PColP  for  remote  display 

A  significant  impact  in  most  computing 
environments  is  the  cost,  time  and  effort  associated 
with  Information  Assurance.  As  noted  in  a  previous 
I/ITSEC  paper,  “The  largest  challenge  affecting  the 
functionality  of  SO  A,  cloud  computing,  and 
virtualization  is  security.  Security  practices  are 
necessary  in  order  to  protect  systems  from  viruses 
and  data  leakage;  however,  security  degrades  system 
performance.”  (Lanman  et  al.) 

With  virtualization  technology,  the  IA  burden  can  be 
reduced  or  simplified.  AWSIM  is  GOTS  software, 
and  came  secured  with  a  patch  from  the  Air  Force 
and  a  Security  Compliance  Guide  for  the  necessary 
manual  steps.  The  ICSF  clients  were  also  GOTS, 
with  lock-down  procedures  integrated  into  the 
installation  procedures.  Once  one  ICSF  client  was 
created,  the  duplicates  inherited  the  IA  lock-down 
profile.  In  the  case  of  the  software  for  the  BCS  and 
RTAP  components,  they  were  considered  GOTS 
under  agreement  with  the  Prime  Contractor,  Thales 
Raytheon.  The  BCS  components  took  considerable 
effort  to  lock-down.  However,  the  virtual  RTAP 
machines  inherited  a  locked-down  IA  profile  from  its 
physical  twin.  The  Common  Boundary  System  was 
GOTS,  and  was  installed  as  an  application  on  a 
locked-down  Windows  VM. 

The  thin  clients  were  left  as  pre-configured  from  the 
manufacturer  since  they  always  return  to  that  state 
after  being  powered  down.  The  only  changes  were  to 
install  the  latest  patch  version  of  Adobe  Flash 
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Player™,  assign  a  fixed  IP  address,  and  new 
Username  and  Password. 


SUMMARY 

NORAD  and  USNORTHCOM  crew  training  has 
historically  been  hindered  by  an  inability  to  conduct 
dynamic  training  and  exercises  on  all  of  their 
Command  and  Control  systems.  The  Joint  Training 
and  Exercise  Directorate  had  to  redirect  its  training 
and  exercise  enhancement  approach  when  N-NC 
changed  the  priority  of  C2  systems  used  to  conduct 
the  missions. 

We  leveraged  server  and  desktop  virtualization 
technology  to  create  an  affordable  virtual 
environment  mimicking  the  operational  systems  and 
crew’s  workstations.  The  server  virtualization 
process  created  functionally  equivalent  virtual 
machines  with  the  authentic  end-to-end  processing 
and  message  exchanges. 

Desktop  virtualization  technology  allowed  the  crews 
to  view  and  manage  multiple  client  VMs  without 
significantly  affecting  their  existing  workspace.  This 
enabled  the  Air  Domain  crews  to  “Train  as  you 
Fight.” 

The  specifics  detailed  in  this  paper  describe  the 
savings  N-NC  realized  in  overcoming  C2  system 
training  shortfalls.  In  general,  the  versatility  of 
virtualization  can  provide  similar  savings  for  other 
operational,  testing,  training,  or  office  uses. 
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